Run test262 tests in CI #408
Open
+32
−3
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
nickva
force-pushed
the
run-test262-in-ci
branch
12 times, most recently
from
8ee7286 to
24e9e37
Compare
nickva
force-pushed
the
run-test262-in-ci
branch
4 times, most recently
from
9c0da0d to
6960dd5
Compare
nickva
force-pushed
the
run-test262-in-ci
branch
2 times, most recently
from
ab51af9 to
7ca7dc7
Compare
nickva
force-pushed
the
run-test262-in-ci
branch
2 times, most recently
from
4a07bd7 to
44d52ea
Compare
nickva
added a commit
to apache/couchdb
that referenced
this pull request
Jun 15, 2025
Test262 are 70k+ Javascript conformance tests [1]. QuickJS has its own test runner but unfortunately there is not automatic bootstraping and they run by hand. There is an upstream PR [2] to add these tests to run there but until that happens we can patch our own QuickJS with. This particular commit doesn't run the tests just patches the necessary updates for bootstrapping and some timeout tweaks to fix some flakes. [1] https://github.com/tc39/test262 [2] bellard/quickjs#408
nickva
added a commit
to apache/couchdb
that referenced
this pull request
Jun 15, 2025
Test262 are 70k+ Javascript conformance tests [1]. There is an upstream PR [2] to run these in CI there but until then can patch our own QuickJS and run them in our CI. One of the patches we have `04-test262-errors.patch` patches one of the exception we expect to get since modified the parser to accept the `function` expression syntax. [1] https://github.com/tc39/test262 [2] bellard/quickjs#408
nickva
added a commit
to apache/couchdb
that referenced
this pull request
Jun 15, 2025
Test262 are 70k+ Javascript conformance tests [1]. There is an upstream PR [2] to run these in CI there but until then can patch our own QuickJS and run them in our CI. One of the patches we have `04-test262-errors.patch` patches one of the exception we expect to get since modified the parser to accept the `function` expression syntax. [1] https://github.com/tc39/test262 [2] bellard/quickjs#408
nickva
added a commit
to apache/couchdb
that referenced
this pull request
Jun 15, 2025
Test262 are 70k+ Javascript conformance tests [1]. There is an upstream PR [2] to run these in CI there but until then can patch our own QuickJS and run them in our CI. One of the patches we have `04-test262-errors.patch` patches one of the exception we expect to get since modified the parser to accept the `function` expression syntax. [1] https://github.com/tc39/test262 [2] bellard/quickjs#408
nickva
force-pushed
the
run-test262-in-ci
branch
2 times, most recently
from
0ced3d2 to
17475ee
Compare
nickva
added a commit
to apache/couchdb
that referenced
this pull request
Jun 20, 2025
Test262 are 70k+ Javascript conformance tests [1]. There is an upstream PR [2] to run these in CI there but until then we can patch our own QuickJS and run them in our CI. One of the patches we have `04-test262-errors.patch` patches one of the exception we expect to get since modified the parser to accept the `function` expression syntax. [1] https://github.com/tc39/test262 [2] bellard/quickjs#408
nickva
added a commit
to apache/couchdb
that referenced
this pull request
Jun 20, 2025
Test262 are 70k+ Javascript conformance tests [1]. There is an upstream PR [2] to run these in CI there but until then we can patch our own QuickJS and run them in our CI. One of the patches we have `04-test262-errors.patch` patches one of the exception we expect to get since modified the parser to accept the `function` expression syntax. For now run the test in the full CI only for Linux and skip s390x. Some architectures/os combinations show various discrepancies around TypedArrays or rounding errors. [1] https://github.com/tc39/test262 [2] bellard/quickjs#408
nickva
added a commit
to apache/couchdb
that referenced
this pull request
Jun 21, 2025
Test262 are 70k+ Javascript conformance tests [1]. There is an upstream PR [2] to run these in CI there but until then we can patch our own QuickJS and run them in our CI. One of the patches we have `04-test262-errors.patch` patches one of the exception we expect to get since modified the parser to accept the `function` expression syntax. For now run the test in the full CI only for Linux and skip s390x. Some architectures/os combinations show various discrepancies around TypedArrays or rounding errors. [1] https://github.com/tc39/test262 [2] bellard/quickjs#408
nickva
added a commit
to apache/couchdb
that referenced
this pull request
Jun 21, 2025
Test262 are 70k+ Javascript conformance tests [1]. There is an upstream PR [2] to run these in CI there but until then we can patch our own QuickJS and run them in our CI. One of the patches we have `04-test262-errors.patch` patches one of the exception we expect to get since modified the parser to accept the `function` expression syntax. For now run the test in the full CI only for Linux and skip s390x. Some architectures/os combinations show various discrepancies around TypedArrays or rounding errors. [1] https://github.com/tc39/test262 [2] bellard/quickjs#408
nickva
force-pushed
the
run-test262-in-ci
branch
2 times, most recently
from
2b07a0b to
11325a5
Compare
nickva
force-pushed
the
run-test262-in-ci
branch
2 times, most recently
from
8c30777 to
6f8b450
Compare
|
@bellard any interest in having the CI run test262 tests? I noticed you keep the test262 commit sha in the TODO file but I put it in the Makefile, not sure my approach is good. Maybe it could be a separate file? The idea is to have repeatability if the CI job is re-run and test262 is updated in the meantime. |
|
Your approach looks OK. I'll remove the test262 commit sha in TODO ASAP. |
nickva
force-pushed
the
run-test262-in-ci
branch
from
6f8b450 to
3c38f29
Compare
|
Updated with latest test262 from yesterday (2025-09-14). Needed |
nickva
force-pushed
the
run-test262-in-ci
branch
from
3c38f29 to
8b43bae
Compare
|
Updated it with the latest test262 and it seems to be all green 
What do you think, Fabrice, does it need any more changes or tweaks? |
nickva
force-pushed
the
run-test262-in-ci
branch
4 times, most recently
from
42e7c2b to
19d8e52
Compare
Run test262 tests for Linux, macOS and Cosmopolitan Add a Makefile `test2-bootstrap` helper to clone and patch test262 tests at a particular commit. Running with the latest commit noticed a few tests were failing and added them to the errors list. The CI environment, especially for macOS, was a bit more unpredictable and was failing in atomics tests due to lower timeouts in the `atomicsHelper` so double the timeouts. The time didn't worsen too much. It still takes about 2-3 min.
nickva
force-pushed
the
run-test262-in-ci
branch
from
19d8e52 to
0df111b
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Run test262 tests for Linux, macOS and Cosmopolitan
Add a Makefile
test2-bootstraphelper to clone and patch test262 tests at a particular commit. Running with the latest commit noticed a few tests were failing and added them to the errors list.The CI environment, especially for macOS, was a bit more unpredictable and was failing in atomics tests due to lower timeouts in the
atomicsHelperso double the timeouts. The time didn't worsen too much. It still takes about 2-3 min.